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Abstract Body 


Background / Context: Adolescents in the United States and their educators face an enormous 
challenge with respect to reading comprehension. College and career readiness standards 
outlined in the Common Core State Standards Initiative (2012) place increased emphasis on 
preparing students to read complex text across a range of content areas. At issue is how to 
develop the necessary skills to be able to read the texts required of college classes and literacy- 
demanding occupations when fewer than 35% of students in the secondary grades read 
proficiently (U.S. Department of Education, 2011). Compounding the challenge is the reality 
that secondary-level courses are largely focused on disciplinary content where reading skills are 
prerequisites and therefore, not an instructional priority (Kamil et ah, 2008; Kennedy & Ihle, 
2012). The increased demands to read and learn from complex text in combination with the 
reading proficiency levels of today’s adolescents bring into sharp relief the academic chasm that 
exists in secondary classes (Eason, Goldberg, Young, Geist, & Cutting, 2012) and the need to 
engage content-area teachers in finding a solution to closing it. 

An emerging body of research has identified promising practices to promote reading 
comprehension among adolescents (Biancarosa & Snow, 2006; Boardman et ah, 2008; Edmonds 
et ah, 2009). Dole, Nokes, and Drits (2009) categorized comprehension practices into two 
categories: teaching practices and self-regulated learning strategies (i.e., strategies students use to 
gain understanding or knowledge). Eindings from meta-analyses (Davis, 2010; Edmonds, 2009) 
suggest that multicomponent comprehension intervention holds promise. Nevertheless, to date, 
only a limited number of studies have examined the combination of teacher-directed and student- 
regulated practices conducted in secondary content-area classrooms. 

Purpose / Objective / Research Question / Focus of Study: This study expands this evidence 
base by comparing a theoretically developed set of practices designed to develop knowledge and 
student-regulated text processing. Extending findings from recent meta-analyses of 
comprehension strategy instruction (Davis, 2010; Edmonds et ah, 2009), we investigated effects 
of integrating teacher-directed knowledge-building and student-regulated comprehension 
practices on 7-10* grade students’ reading comprehension. We targeted three primary research 
questions: (a) What are the statistical and practical effects of knowledge-building and student- 
regulated practices on 7th to lOth-grade students’ reading comprehension? 

(b) What is the relationship between dimensions of fidelity of implementation (i.e., adherence, 
quality of instruction, and program differentiation) and reading comprehension? (c) Do 
knowledge-building and student-regulated practices differentially benefit subgroups of students 
with lower entry-level comprehension performance? 

Setting: The study took place in three middle schools and three high schools from three districts 
in south/central Texas. School enrollments ranged from 353-1,219. All participating schools 
received Title I funding and had high percentages (59%-76%) of students who qualified for free 
or reduced-price lunch. The districts educated a diverse group of students, with 76.6% 
economically disadvantaged and 6.5% Eimited English Proficiency (EEP) in District A, 76.2% 
economically disadvantaged and 15.5% EEP in District B, and 71.8% economically 
disadvantaged and 17.6% EEP in District C. 
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Population / Participants / Subjects: Seventeen 7th- to lOth-grade English language arts 
teachers (3 males and 14 females) participated in this study. All teachers instructed at least two 
sections of English language arts/reading daily (M = 3.82, Mdn = 4). Teaching experience 
ranged from 0-23 years with a mean of 5.66 years (SD = 6.18). All teachers held bachelor’s 
degrees, three had master’s degrees, and three had educational specialist or doctoral degrees. Of 
the 17 teachers, 14 held English language arts certifications and 7 held multiple certifications. 

Students (A = 921) from participating classes were randomly assigned within teacher to 
intervention (n = 489; 36 classes) or typical practice conditions (n = 432; 29 classes). Student 
demographics for each condition are described in Table 1 (please insert Table 1 here). Group 
equivalence was evaluated using independent-sample t-tests for continuous variables (e.g., age) 
and chi-square tests for categorical variables (e.g., gender, ethnicity). 

Intervention / Program / Practice: The intervention involved a routine of teacher-directed and 
student-regulated practices. To increase curricular fit, we aligned practices with the content and 
sequence in which text genres were introduced in each school. Teachers identified short stories 
and expository text as appropriate targets for comprehension intervention and recommended the 
intervention be restricted to the fall semester to avoid spring state assessment conflicts. Teachers 
identified exemplar texts for each grade (e.g. The Most Dangerous Game by Richard Connell 
and The Monkey’s Paw by W.W. Jacobs). Organized in three phases (see Table 2), practices 
were introduced using short stories. The routine and instructional practices were then transferred 
to expository text, and finally to teacher-selected texts. Phases 1 and 2 provided structured 
activities and recommended language; Phase 3 was designed for generalization. Practices were 
implemented in the fall semester. Teachers were asked to implement practices three days per 
week for 50 minutes; the teachers determined their schedule of implementation within the 
semester. 

The three primary components comprised teacher-directed Text Set-up, partner-regulated Text 
Analysis and Dialogue, and teacher- and student-directed Text Synthesis. Each component was 
used with a text (e.g., short story, expository selection), and the amount of time per component 
varied by text length and complexity. Narrative and informational components used a common 
set of practices; however, question types, graphic organizers, and assignments differed to reflect 
text structure and topic. The entire cycle was designed to take 120-150 minutes, distributed 
across three days (please insert Table 2 here) 

Professional development sessions were distributed over the fall semester and included face-to- 
face researcher-directed sessions, small-group meetings and individual coaching. Specifically, 
prior to the start of the school year, research team members conducted a full day of professional 
development that focused on applying intervention practices to short stories. Two research staff 
worked with teachers over the course of the semester, meeting with them individually and in 
grade-level teams. 

Research Design: A within-teacher, randomized block design was used to study the effects of 
intervention on 7th- to lOth-grade students’ distal and proximal reading comprehension. Each 
teacher’s English language arts classes were randomly assigned to either intervention or typical 
practice. We examined the impact of fidelity of implementation variables, including adherence. 
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global ratings of quality and classroom management, and program differentiation. Finally, we 
were interested in determining whether there was differential impact of the intervention by grade 
and students’ entry-level comprehension. 

Data Collection and Analysis: Distal Measure of Reading Comprehension (pre/posttest). 

Gates MacGinitie Reading Test, 4th ed. (GMRT-4; MaeGinitie, MacGinitie, Maria, Dreyer, & 
Hughes, 2000). The Gates MaeGinitie Reading Test is a timed, group-administered assessment 
of reading eomprehension. It consists of informational and narrative passages ranging in length 
from 3 to 15 sentenees. Students read eaeh passage silently and then answered 3-6 multiple- 
choice questions, inereasing in diffieulty; there is a 3 5 -minute time limit. Internal consistency 
reliability ranges from .90 to .95, and alternate-form reliability is reported as .80 to .83. 

Researcher-adapted proximal comprehension measure (pre/posttest). Two passages and 
question sets from the Adolescent Literacy Inventory (Brozo & Afflerbaoh, 2011) were adapted 
to assess proximal comprehension. The first passage was a narrative text followed by 15 
multiple-ehoiee questions. The second passage was expository with 16 multiple-ohoiee 
questions. Modifieations to the existing question protoeols included the addition of multiple- 
choice answer options to existing open-ended questions, the addition of a fourth answer choice to 
existing multiple-choice questions, and new multiple-ehoiee questions. Question types for both 
passages included literal (i.e., information presented explicitly in the text), inferential (i.e., 
information not explicitly stated in the text, or requiring integration of information from the text 
or prior knowledge), or voeabulary-based questions. The measure was untimed and was group- 
administered during a 50 minute session. Students earned 1 point for each correct answer with a 
maximum score of 3 1 . The reliability of the measure was .80 (Cronbaeh’s alpha). 

Fidelity of implementation. Beeause teachers served in both conditions, we examined multiple 
dimensions of fidelity to evaluate implementation of intervention proeedures and identify any 
contamination from intervention to typieal praetiee classrooms. Fidelity measures included 
intervention adherence by component (intervention group only), global ratings of instruction 
(i.e., quality and classroom management), and program differentiation of instruetional praetiees 
(Dane & Schneider, 1998). 

Analyses. We used multilevel modeling to examine statistically significant differences between 
the intervention and the comparison condition. Three-level models were used to control for the 
nested nature of our data, with 921 students nested within 65 classes, whieh were nested within 
17 teaehers. The models were fit using MLwIN (MLwIN 2.20; Rasbash, Charlton, Browne, 
Healy, & Cameron, 2009). Outeome measures included the GMRT-4 and the researeher-adapted 
narrative and expository assessments. Student pretest scores on the related measures (e.g., 
GMRT-4 pretest for the GMRT-4 outeome variable) were used as student-level covariates to 
control for entry-level scores. A dummy-eoded variable representing intervention and 
eomparison condition was entered as a classroom-level variable, with 0 representing the 
comparison condition and 1 representing the intervention condition. Subsequent analyses used 
the same outcome variables and examined differences between conditions using covariates 
ineluding grade level, reading status, and fidelity. The Hedge’s g effect size (2007) was 
calculated for all models. The Benjamini-Hoehberg (Benjamini & Hoehberg, 1995) eorreetion 
was used to control inflated family-wise error rate due to comparisons of multiple outcomes. 
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Findings / Results: Intervention effects. There are no statistieally signifieant differenees 
between conditions on any posttest measure. Hedges’ g effect sizes range from -.01 (GMRT-4) 
to .03 (expository passage) where positive scores indicate the intervention condition performed 
better than comparison. There is, however, a statistically significant effect for time on the 
GMRT-4, with both groups evidencing higher performance at posttest than pretest (t = 11.91 ,P< 
.000). Given that pre- and posttest norms are adjusted for time, gains in performance were 
considered practically important with a standardized mean difference effect size equal to .46. 

The mean performance for both groups was around the 32th percentile at pretest and the 42nd 
percentile at posttest, with an average mean difference in standard scores of 4.35 for the 
intervention group and 3.84 for the comparison group. Significant pretest to posttest growth is 
also evident on the narrative (t = 2.21, p = .027) and expository (t = 5.63, p < .000) proximal 
measures. The standardized mean difference effect sizes for time are .09 for the narrative 
passage and .22 for the expository passage. 

Fidelity effects. Mean adherence ratings averaged across phases indicate modest implementation 
with an average rating of 3.03 (SD 1.11). This score indicates that half of the intervention 
components were implemented during sessions observed. Global implementation across phases 
shows a slight nonsignificant decline (r = -.1.44, p = .281) in implementation over the course of 
the intervention. On average, teachers had higher overall adherence scores earlier in the 
intervention when using researcher-prepared lessons for short stories (M = 3.27). Mean 
adherence for the expository text phase was 3.05 and for generalization lessons, 2.86. Simple 
correlations between fidelity and student achievement outcomes do not exceed -0.176. 

Findings from observations of instructional practices (not specific to intervention) indicate that 
teachers in both conditions employed several similar strategies to promote reading 
comprehension across all three phases of the intervention. For example, activating background 
knowledge was observed between 45 to 77% percent of typical practice conditions, in 
comparison, this strategy was observed between 55 to 81% of intervention conditions. 
Noteworthy among these findings is that teacher-regulated knowledge-building practices 
(background knowledge and vocabulary) looked different in the earlier phases of the study (45% 
typical practice and 79-86% intervention) but grew more alike in the generalization phase (62- 
77% typical practice; 81-86% intervention). 

Differential effects. Results of the main effect for entry-level reading ability are mixed. On the 
GRMT-4, students who entered below the 15th percentile demonstrate more improvement than 
students who entered above that point. The opposite is true for both the narrative and expository 
passages. On the expository passage, students above the 15th percentile perform statistically 
significantly better than students below the 15th percentile (Hedges’ g = .23). Findings from 
interaction analyses suggest that higher-performing readers benefitted more than their lower- 
performing peers from the intervention that emphasized student-regulated practices as indicated 
by a positive Hedges’ g for the interaction term on all measures. 

Conclusions: This study was conducted to advance our understanding of instructional methods 
to promote secondary students’ comprehension of texts read in English language arts classes. 

The intervention that combined teacher-directed knowledge-building and student-regulated 
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practices did not differentially impact 7th-10th grade students’ comprehension on proximal or 
distal measures of reading eomprehension over English language arts teaehers’ typieal practice. 
Effect sizes that ranged from - .01 to .03 failed to approximate those in the .36 - .89 range 
doeumented in prior researeh synthesis (Davis, 2010; Edmonds, 2009). Perhaps the most 
promising of findings is that students in both eonditions evidenced statistically significant growth 
from pretest to posttest on proximal and standardized distal measures of reading eomprehension. 
Our hypothesis that students with lower-entry level comprehension would benefit more than their 
higher-achieving peers from the integration of teaeher-directed and student-regulated practices 
was not confirmed. In faet, students with higher-entry level reading eomprehension benefitted 
more on narrative and expository eomprehension than peers with lower entry-level 
eomprehension. 

There are three plausible explanations for the finding of statistically significant but 
nondifferential growth of students in the intervention and typieal praetiee eonditions. Perhaps 
the most probable is that the general comprehension praetices used in typieal praetiee were as 
effective as those speeific to the intervention. A seeond logical explanation for the pattern of 
findings resides in whether there is a treatment adherence threshold neeessary to produce 
differential performanee. Our findings validate eoneems raised by prior researehers about 
implementation variability and fidelity of implementation at the seeondary level (Klingner & 
Vaughn; 1996; Wharton-MeDonald & Swiger, 2009). Third, differential patterns of response by 
students with different levels of reading proficieney may have masked important patterns. We 
were interested in whether interventions that plaeed inereasing emphasis on student regulation 
would have differential impaet based on students’ entry-level reading profieieney. Results of 
proficiency level analyses yielded nonstatistically significant but nonetheless complex effect size 
findings that varied by measure. On the GMRT-4, students whose entry-level scores were below 
the 15th pereentile grew more than students above the 15th pereentile with the opposite pattern 
oeeurring on the narrative and expository proximal measures. Effeets sizes from interaetion 
analyses on proximal measures indieated that students in the treatment group with higher reading 
scores benefitted more than students with seores below the 15th percentile with statistically 
significant effects on the expository measure. 

Limitations, bindings of this study must be plaeed in the context of several limitations. Eirst, 
the study was eonducted in sehool districts with exceptionally high pereentages of students with 
low levels of reading profieieney. Thus, further researeh is needed to determine whether 
findings generalize to sehools with higher pereentages of proficient readers. Seeond, beeause 
teaehers taught both intervention and typieal practice classes, they were exposed to and to some 
degree transferred intervention practices to their typical instruction. It is exceedingly difficult to 
populate randomized controlled trials in seeondary sehools, and teaehers serving as their own 
control has proven to be a viable design with limited evidence of eontamination. Einally, the 
proportion of students with signifieantly low reading comprehension skills was unexpected and it 
would be extremely valuable to understand the nature of their reading eomprehension 
diffieulties. 
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Table 1 

Descriptive Student Demographics by Condition 

Intervention Comparison 

{n = 489) {n = 432) 


Variable 

n 

(%) 

n 

(%) 

Gender 

Male 

226 

46.7 

206 

48.2 

Female 

258 

53.3 

221 

51.8 

Ethnicity 

Asian 

0 

0.0 

1 

0.2 

American Indian or Alaska Native 

1 

0.2 

2 

0.5 

Black or African-American 

160 

33.2 

123 

28.5 

Hispanic or Latino 

176 

36.5 

148 

34.7 

White 

145 

29.7 

151 

35.3 

Identified for special education 

17 

4.6 

22 

6.6 

Free or reduced lunch 

339 

70.5 

299 

70.5 

English language learner 

31 

6.5 

29 

6.8 
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Table 2 


Intervention Emphasis and Timeline by Phase 


Lesson 

Numbers 

Type of Lessons 

Suggested Timeline 

Days of 
Instruction 


Phase 1: 

Short Stories 


1-10 

11-13 

Short story introduetory lessons 
Teaeher ehoiee of short story 

First Six Weeks: Weeks 4 & 5 
First Six Weeks: Week 6 

13 days 


Phase 2: Informational Text 


14-20 

Informational text lessons 

Second Six Weeks: Weeks 2-3 

10 days 

21-23 

Teaeher ehoiee of informational 
text 


24-35 

Phase 3: Generalization of Practices 

4 3 -day cycles of teacher choice 
of text (short story, 

informational text, novel, or Second Six Weeks: Weeks 4-5 

combination of these text types; & Third Six Weeks: Weeks 1-2 

preferably linking a narrative 
and informational text) 

12 days 
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